Interface method and system for use with computer directed assembly and manufacturing

ABSTRACT

A computer-implemented user interface displayed on a display of a computer directed assembly workstation is provided for interacting with a sequence of assembly instructions. In some embodiments, an in-use view area is displayed on the user interface providing a “live” image of an item the assembler is currently working on. This in-use view area generally involves taking videos of tools, materials, matrix trays with materials, and/or components of the final product as they are manipulated by the assembler. The user interface further displays a preassembled view area providing an image of at least one preassembled item provided as an example of a properly assembled item. This example helps the assembler process each assembly instruction with greater accuracy and speed. This preassembled view area may include other products previously assembled and useful to the assembler performing the assembly task including images and videos of materials, components, and/or matrix trays with materials.

(1) CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.61/493,958, filed Jun. 6, 2011, entitled, “MANUFACTURING VERIFICATIONMETHOD AND SYSTEM” by Balbir S. RATAUL, assigned to the assignee of thisapplication and incorporated by reference herein for all purposes. Thesubject matter of this application further relates to the subject matterof the following commonly assigned applications being filed on the sameday as the present application: (1) U.S. Ser. No. 13/______, AttorneyDocket 00163-000100000 entitled, “COMPUTER DIRECTED ASSEMBLY METHOD ANDSYSTEM FOR MANUFACTURING”; (2) U.S. Ser. No. 13/______, Attorney Docket00163-000200000 entitled, “VERIFICATION METHODS AND SYSTEMS FOR USE INCOMPUTER DIRECTED ASSEMBLY AND MANUFACTURE”; (3) U.S. Ser. No.13/______, Attorney Docket 00163-000400000 entitled, “SYSTEM AND METHODFOR MANAGING TOOL CALIBRATION IN COMPUTER DIRECTED ASSEMBLY ANDMANUFACTURING”; (4) U.S. Ser. No. 13/______, Attorney Docket00163-000500000 entitled, “INTERFACE METHOD AND SYSTEM FOR USE WITHCOMPUTER DIRECTED ASSEMBLY AND MANUFACTURING”; (5) U.S. Ser. No.13/______, Attorney Docket 00163-000600000 entitled, “TRAINING ENSURANCEMETHOD AND SYSTEM FOR COMPUTER DIRECTED ASSEMBLY AND MANUFACTURING”; (6)U.S. Ser. No. 13/______, Attorney Docket 00163-000700000 entitled,“RESOURCE SCHEDULING METHOD AND SYSTEM FOR USE WITH COMPUTER DIRECTEDASSEMBLY AND MANUFACTURE”. Each of the above-referenced patentapplications is incorporated by reference herein for all purposes.

(2) TECHNICAL FIELD

The present invention relates to systems and methods used in amanufacturing. More particularly, the present invention relates to auser interface method and corresponding system for using computers toassist in assembling and verifying the assembly of manufacturedproducts.

(3) DESCRIPTION OF THE RELATED ART

A great deal of modern devices and equipment continue to be manufacturedusing manual assembly. Increasingly, complex mechanical, electrical, andelectro-mechanical designs having relatively small dimensions requireskilled and trained assemblers to perform a variety of assembly tasks,some done directly with their hands or with the assistance of a varietyof precision hand tools. This type of manufacturing is often deemedlight manufacturing as it involves applying a certain degree of humanskill and know-how to combine fasteners, connectors, and other materialsin the creation of the final manufactured product. Light manufacturingand manual assembly are also often preferred for smaller production runswhen costs associated with automation and retooling cannot be amortizedover the production run time frame while maintaining profit margins.

To help reduce human error and other mistakes, conventional lightmanufacturing methods incorporate a “Manufacturing Process Instruction”(MPI) document in either a hard-copy form or displayed on a computermonitor that each assembler refers to during the assembly process. TheMPI may be created by a manufacturer to provide specific instructionsfor the assembly of a wide-range of products from computers, householdelectronics, communication equipment, or even sophisticated medicalequipment. In each of these categories, the quality and consistency ofthe final product produced depends on whether the person involved withassembling an assembly or subassemblies actually understands andaccurately follows the instructions within the MPI.

Since the MPI does not actually control the act of assembling products,product quality may be inconsistent or lower than desired. An assemblermay initially follow every step of the MPI document to produce highquality products but later deviate from the MPI instructions and produceproducts with defects or other problems. In another scenario, anassembler may follow his own assembly sequence and, as needed, flipthrough the MPI document as an occasional reference. Products assembledin this latter approach may be of consistent but overall lower qualityif the approach taken by the assembler consistently skips steps or takesunacceptable shortcuts.

The lack of controls and accountability associated with the MPI documentalso makes it difficult to track down and find the source of a problem.This is especially true if there are many sub-assemblies or componentsthat makeup the overall manufactured products. Indeed, checklists may beused in conjunction with the MPI document to query the assembler andverify whether instructions in the MPI document were taken. Once again,the assembler may not answer or inaccurately answer questions in thechecklist thus circumnavigating the quality control checkpointsprovided.

SUMMARY

Aspects of the disclosure provide methods, systems, and computer programproducts for generating computer-implemented assembly instructions toassist an assembler to create a manufactured product. Generally, aproduct designer or engineer develops a product and then creates theassembly instructions to assemble the product. Assemblers use theseassembly instructions to guide the manual assembly of materials,components and other parts into the final product. Typically, theassembler executes these assembly instructions on a specially equippedcomputer directed assembly (CDA) workstation as described herein toensure the products are produced both quickly and with the highestquality.

In some embodiments, a computer-implemented user interface method isdisplayed on a display device of the CDA workstation for interactingwith the sequence of assembly instructions. Some embodiments of the userinterface method display an in-use view area on the user interface thatprovides a “live” image of an item the assembler is currently workingon. This in-use view area generally involves taking videos of tools,materials, matrix trays with materials, and/or components of the finalproduct as they are manipulated by the assembler. In addition, in someembodiments, the user interface method further displays a preassembledview area providing an image of at least one preassembled item providedas an example of properly assembled item. This example helps theassembler process an assembly instruction. In some embodiments, thispreassembled view area may include other products previously assembledand useful to the assembler performing the assembly task includingimages and videos of materials, components, and matrix trays withmaterials.

If tools are to be used, the tool is generally displayed in a tool viewarea on the user interface that provides an image of a tool as specifiedin the sequence of assembly instructions. Video, audio, and/or othertypes of multimedia may accompany the image of the tool as displayed inthe tool view area. In a text assembly instruction area of the userinterface, in some embodiments, a text assembly instructions area isdisplayed that provides text and/or audible instructions derived fromthe sequence of assembly instructions. These text assembly instructionsguide the assembler with specific instructions for using a tool orperforming an assembly instruction in a text and/or audio format andoften may reference images, video, and/or other multimedia displayed tothe user in other portions of the user interface.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a high level block diagram illustrating an exemplarydevelopment process of taking a product from conceptual design tomanufacture in accordance with some embodiments;

FIG. 2 is a schematic block diagram illustrating an exemplary computerdirected assembly workstation for building and verifying assemblies inaccordance with some embodiments;

FIG. 3 is a flow chart diagram associated with creating a sequence ofassembly operations for use on a computerized assembly workstation inaccordance with some embodiments;

FIG. 4 is yet another flow chart diagram for using multimedia assemblydata along with a sequence of assembly operations for guiding the use ofmatrix trays and insertable shims in accordance with some embodiments;

FIG. 5 is another flow chart diagram outlining an exemplary trainingverification operations performed in accordance with some embodimentsprior to providing an assembler authorization to perform an instructionassociated with an assembly operation;

FIG. 6A is a schematic diagram of an exemplary computer implemented userinterface for interacting with a sequence of assembly operations inaccordance with some embodiments;

FIG. 6B is a schematic diagram of an exemplary matrix tray with multipleinsertable shims and a variety of pockets for materials in accordancewith some embodiments;

FIGS. 7A-7B are additional flowchart diagrams outlining exemplaryinteractions associated with performing the assembly operations on acomputer directed assembly workstation in accordance with someembodiments; and

FIG. 8 is a schematic block diagram of an exemplary computer device usedin generating and performing computerized assembly operations inaccordance with some embodiments.

DETAILED DESCRIPTION

In the following detailed description, for purposes of explanation,numerous specific details are set forth to provide a thoroughunderstanding of the various embodiments of the disclosure. Those ofordinary skill in the art will realize that these various embodiments ofthe present invention are illustrative only and are not intended to belimiting in any way. Other embodiments of the present invention willreadily suggest themselves to such skilled persons having the benefit ofthis disclosure.

In addition, for clarity purposes, not all of the routine features ofthe embodiments described herein are shown or described. One of ordinaryskill in the art would readily appreciate that in the development of anysuch actual implementation, numerous implementation-specific decisionsmay be required to achieve specific design objectives. These designobjectives will vary from one implementation to another and from onedeveloper to another. Moreover, it will be appreciated that such adevelopment effort might be complex and time-consuming but wouldnevertheless be a routine engineering undertaking for those of ordinaryskill in the art having the benefit of this disclosure.

A computer directed assembly method and system designed in accordancewith embodiments of the present invention offers many advantages andbenefits, some of these advantages include one or more of the following.Generating the assembly operations for assembling a component may betailored to the address specific assembly requirements in eachparticular design. If an assembly is complex with strict tolerances, thedesigner or manufacturing engineer may decide to incorporate morediscrete assembly operations or steps to ensure the assembly is puttogether correctly and completely. The assembly operations mayincorporate different types of multimedia content including images,videos, text, and audio to convey the specific steps necessary tocomplete an operation accurately and efficiently.

Embodiments described herein maintain control over the assembly processand ensure each step of the assembly is performed in sequence. Assemblyoperations for a component are presented in order and recorded as both ahistory of the component being assembled and the task performed by theassembler. Recording the assembly operations serves to create apermanent record while encouraging the assembler to follow the assemblyroutine. To confirm that a component is being assembled according toplan, some embodiments include verification steps that record images,video, audio, and/or other multimedia data of the component as thecomponent is assembled. In some embodiments, the multimedia data createsa permanent record and a traceable sequence of events available if theassembly record needs later review. In other embodiments, verificationsteps may further include comparing a predetermined image of apreviously assembled component with an image taken of an “in-use”component being assembled. Embodiments may alternatively use imageprocessing routines to perform an immediate and direct comparisonbetween a predetermined image of an assembly and the in-use image of apart or other component “in-use” and being assembled.

FIG. 1 is a high level block diagram illustrating an exemplarydevelopment process for a manufactured product from conceptual design tomanufacture in accordance with some embodiments. This developmentprocess illustrated in FIG. 1 provides one exemplary developmentalprocess where the computer directed assembly processing designed inaccordance with some embodiments can be used. Other developmentprocesses may also be used in other embodiments as well. For example,these other development processes may include greater or fewer stepsthan illustrated in FIG. 1 or may combine several of the steps in FIG. 1together or may expand certain steps in FIG. 1 into additional substeps.

Referring to FIG. 1, the development process for a product generallystarts with a conceptual design and prototype 102. In the earliest stageof product development, a person or team of people have an idea anddecide if the idea can be turned into a product. Next, sketches of theproduct are either drawn by hand or using computer aided design (CAD) orsimilar tools. The design may be tested using simulation tools toquickly determine if the product will produce a desired result under oneor a variety of “what if” scenarios. If the conceptual design andtesting is viable, an actual prototype of the product may be puttogether using readily available components and manual assemblytechniques. Since only one or several prototypes of a product are made,cost and volume production considerations are not generally the highestconcern.

During product design 104, the prototype of a product may change toaccommodate both design and production or requirements. Some designrequirements affecting product design 104 may include aesthetic changesin the shape of the product, a reduced form factor to make the productfit into a smaller or thinner package for production. As part of productdesign 104, availability and costs for components and materials, as wellas the total costs associated with assembling and shipping the finishedproduct should also be considered. In some embodiments, manufactureengineering 106 provides feedback to product designers concerning thecosts of materials and creation of the product as proposed in productdesign 104. In some embodiments, manufacture engineering 106 worksiteratively with product design 104 refining the details and influencingthe direction of the final product to be produced from product design104. For example, people involved with manufacturing engineering 106 mayfind that the materials specified to create the components for theproduct are too expensive or the logistics associated with acquiring thematerials for the product will not meet the projected demand.

Computer directed assembly (CDA) engineering 108, in accordance withsome embodiments, incorporates numerous innovative manufacturing methodsand systems in the manufacture of products while meeting and meetnumerous constraints. In some embodiments, CDA engineering 108 includesmultiple disciplines from industrial engineering, human factors,computer science, tooling and other areas to improve light-assembly ofproducts with increased efficiencies, including an assembly instructionsequence 116, tool usage control 118, material usage 120, assemblyverification 122, and assembler training and verification 114.

In some embodiments, CDA engineering 108 creates an assembly instructionsequence 116 tailored to the specific assembly of each product. Theassembly instruction sequence 116 ensures each product is assembledproperly by providing an assembler a step-by-step sequence of tasks toperform. To make sure the assembler understands how to perform eachassembly instruction, the computer directed assembly workstation 124processing the assembly instruction sequence 116 may present one or moretypes of multimedia data including text, images, video, and audio. Forexample, a user interface running on the computer directed assemblyworkstation 124 may present text statements describing the task to beperformed while corresponding images or videos may illustrate the taskpreviously performed by a trained assembler.

In the area of material usage 120, CDA engineering 108 specifies thelocation of materials in one or more pockets within a matrix tray.Pockets of each matrix tray are filled in advance with the specificmaterials to be used later by the assembler during the assembly process.Specific predetermined images of the matrix tray and the location ofmaterials in each pocket of the tray are specifically referenced bysteps in assembly instruction sequence 116. This approach reduces wastedmaterials as all the materials in the trays should be used as specifiedin assembly instruction sequence 116. Leftover materials or missingmaterials generally may mean that the assembly was not performedcorrectly or certain materials were misplaced or lost.

CDA engineering 108 also includes tool usage and control 118 to makesure the proper tools are used and the assembly is completed accordingto specified tolerances. Tool usage control 118 incorporated in assemblyinstruction sequence 116 selects tools for the assembler, and the use ofthe tool and torque to apply upon a fastener may also be describedand/or illustrated through computer directed assembly workstation 124.In accordance with some embodiments, computer directed assemblyworkstation 124 may also use tool usage control 118 combined withsensors embedded in the tools as a basis for determining if the toolsneed to be replaced or recalibrated. Power tools not replaced orcalibrated as specified in tool usage control 118 cannot be used forfurther assembly tasks on computer directed assembly workstation 124.

Assembly training and verification 114 is another novel component of CDAengineering 108 in accordance with some embodiments. As new products andcorresponding assembly methods are created, assemblers must make surethey are trained and capable of performing the assembly tasks bothefficiently and accurately. CDA engineering 108 incorporates assemblytraining and verification 114 as part of a database associating eachassembler's skills with specific assembly tasks used to manufacturecertain products. A manufacture engineer uses assembly training andverification 114 within CDA engineering 108 to define a prerequisitetraining for each step in the assembly training sequence 116. Theprerequisite training is required before an assembler is allowed toperform one or several tasks in the assembly of the product. Inaccordance with assembly instruction sequence 116, an assembler lackingproper training indicated by database records in ERP systems 112 aredeemed to lack the proper training and skill and cannot continue with anassembly.

As a further control on quality, assembly verification 122 checks on theaccuracy of tasks performed from assembly instruction sequence 116 inaccordance with some embodiments. The manufacture engineer mayincorporate assembly verification 122 through CDA engineering 108 forsome or all tasks performed by the assembler; more verificationgenerally improves the quality but may increase the time to assemble aproduct. In some embodiments, assembly verification 122 executessoftware instructions on the computer directed assembly workstation 124instructing the assembler to identify where a material has been attachedto a product or component and then take a photo or video of the result.Generally assembly verification 122 encourages an assembler to work moreaccurately as videos, images, and/or other records of assembling theproduct are kept as a permanent record of the assembler's work historyand stored in ERP systems 112 or elsewhere.

Once a product has been assembled into a manufactured product 126,manufactured product 126 is delivered to customers 128 through normaldelivery and shipping channels. In accordance with some embodiments, ERPsystems 112 are enhanced with additional details on the assembly of eachproduct through methods and systems associated with CDA engineering 108.In some embodiments, records and billing associated with supply chain130 are updated to reflect the materials used in the manufacture of aproduct. Specific materials taken from the supply chain 130 may also beidentified, including details such as serial numbers and data ofmanufacture, and stored as part of the assembly records in ERP systems112. Accordingly, ERP systems 112 and other databases enhanced inaccordance with the various embodiments are of particular value toindustries requiring high quality products with detailed records andtraceability to specific materials and their assembly into products. Forexample, detailed information tracking materials and their assembly isuseful in medical, military, space, aeronautical, and other industriesusing products that impact health and/or safety.

Referring to FIG. 2, a schematic block diagram illustrates an exemplarycomputer directed assembly workstation 124 for building and verifyingassemblies in accordance with some embodiments. Computer directedassembly workstation 124 is the same as referenced in FIG. 1 except FIG.2 provides additional details of one configuration consistent with oneembodiment. Accordingly, in some embodiments workstation 124 includes aworkbench 202, an in-use component 204 being assembled into a product, acomputer device 206, a wand 208, a keyboard 210, a scanner device 212, adisplay device 214, a first camera 216, a second camera 218, a headset234, a third camera 232, a matrix tray 220 for holding materials, anetwork 222 such as the Internet or an intranet, and a networkaccessible server 224 with databases 226 including ERP databases.Computer directed assembly (CDA) software (not shown) in accordance withembodiments executes, on computer device 206 orchestrating theinteraction between the assembler and the aforementioned peripheralswhile assembling in-use component 204 into a product. While someembodiments as illustrated in FIG. 2 use wired connections betweencomputer device 206 and peripherals such as wand 208, keyboard 210,scanner device 212, display device 214, first camera 216, and secondcamera 218, third camera 232, headset 234 other embodiments may usewireless connections between one or more of these aforementionedperipherals and computer device 206 as they can be positioned moreeasily around workbench 202 and less likely to interfere with theoverall working area used for assembly.

Workbench 202 provides a surface or area that an assembler 228 uses whenperforming assembly related tasks in conjunction with in-use component204. In some embodiments, assembler 228 may place in-use component 204directly upon workbench 202 especially if the in-use component isrelatively small, lightweight, or both. In alternative embodiments,assembler 228 may place in-use component 204 next to or adjacent toworkbench 202 especially if the in-use component being worked upon orfinal product is larger, oversized, and/or too heavy for the workbench202.

Assembler 228 may use a number of different tools or data gatheringperipherals connected to computer device 206 when operating aworkstation 124 in accordance with some embodiments. In one embodiment,assembler 228 may move wand 208 over one or more areas of in-usecomponent 204 to identify the area of the component being assembled orreceiving various materials. In some embodiments, wand 208 may beapproximately ½ inch in diameter and 12 to 14 inches in length and havean identifiable colored tip. Wand 208 may be equipped with differentcolored light emitting diodes (LEDs) and operatively coupled to receivesignals and power from computer device 206 that drive the brightness andcolors emitted from the LEDs. Further embodiments of wand 208 with LEDsmay be battery powered with rechargeable batteries and wirelesslycontrolled from computer device 206 using Bluetooth, WiFi, or othersuitable technology. Alternatively, the color of the tip of wand 208 maybe painted, dyed, or set using different colored caps and functionwithout power or control signals and thus not required to be connectedwith computer device 206.

As assembler 228 moves wand 208, CDA software may cause first camera 216and second camera 218 to take one or more images or videos of theposition of wand 208 relative to in-use component 204. First camera 216and second camera 218 are positioned to capture images along differentaxes of the plane of workbench 202 however additional cameras may takeoverhead images looking down on workbench or from other perspectives ofworkbench 202. Third camera 232 may be a handheld, lightweight, wired orwireless device that assembler 228 can position as needed to take imagesof in-use component 204 being worked upon. These cameras may beindustrial or consumer grade cameras with motorized lens havingmotorized zoom and motorized iris components in their lens andcontrolled through a lens controller (not shown) driving by computerdevice 206. Taking images from cameras at multiple view points gives abetter understanding of how the assembly took place. This is useful tolater verify whether an assembly was done correctly or incorrectly. Insome embodiments, assembler 228 positions wand 208 in response to one ormore assembly instructions from CDA software on workstation 124requesting the assembler to identify a recent task or operationperformed. Computer device 206 receives images and videos from firstcamera 216 and/or second camera 218 creating a permanent visual recordof the assembly performed near the user specified portion of in-usecomponent 204 indicated with the tip of wand 208.

CDA software further includes a user interface presented on displaydevice 214 and capable, among its numerous features, of showing imagesand videos taken with first camera 216 and/or second camera 218 to theassembler 228. Depending on the color scheme of in-use component 204,assembly instructions may specify that the tip of wand 208 be set to acontrasting color to aid in processing images taken with first camera216 and/or second camera 218. If wand 208 has LED lights, assemblyinstructions executed on computer device 206 may automatically changethe lights on the wand's tip to a predetermined color that providesgreater contrast against the color scheme of in-use component 204. Forexample, if in-use component 204 is a printer circuit board with apredominantly green color scheme, some embodiments may change the LEDlights of wand 208 to a red or yellow color to make wand 208 moreidentifiable. Alternative embodiments of wand 208 may work without LEDsand electronics, instead of lights the assembler 228 may be instructedto place different colored caps over the tip of wand 208 to betteridentify the wand position relative to in-use component 204.

In some embodiments, assembler 228 may use a keyboard 210 to send textdata and communicate with computer device 206, embodiments of the CDAsoftware records these communications along with other information usedduring the assembly. Text data received on computer device 206 fromkeyboard 210 may confirm data or respond to questions posed from one ormore assembly instructions during assembly of a product. Alternatively,keyboard 210 can be used by assembler 228 to control the overalloperation of workstation 124 including pausing, starting, or stoppingthe assembly operations driven by assembly instructions executed onworkstation 124. In some embodiments, barcode scanner 212 operativelycoupled to computer device 206 is used by assembler 228 for scanning barcodes, this is useful for specifically identifying materials,components, and other products used during the assembly operations.

In some embodiments, materials, pieces, or components used by assembler228 during assembly are placed in one or more matrix trays 220. Eachmatrix tray 220 can be customized and transformed to hold a differentquantity of materials depending on the number of shim inserts and theshape and number of the pockets within each of the shim inserts. Theexact number and type of shim inserts inserted in matrix tray 220 dependon the materials being used during assembly and how the assemblyinstructions are setup. In some embodiments, assembly instructionsspecify the numerosity and arrangement of these shims in each matrixtray. During assembly, the instructions identify the matrix tray as wellas the particular pocket within the matrix tray that should hold aparticular material. Referring to FIG. 2, an exemplary matrix tray 220may receive up to four shim inserts with pockets ranging from one toeight pockets for each shim. Materials are loaded into matrix trays 220in advance with the specific materials required for assembling acomponent and product. Alternate embodiments may include matrix trays,for example, larger than matrix tray 220, capable of receiving a greaternumber of shim inserts with a greater number of pockets and a range ofdifferent sizes. Matrix tray 220 and insertable shims in FIG. 2 may beidentified by assembler 228 during assembly using scanner 212 to scanand “read” a bar code affixed to or associated with matrix tray 220and/or corresponding shims.

One or several power tools connected to computer device 206 may be usedby assembler 228 to assemble a product. In some embodiments, a powertool such as a screw gun 230 provides operational information tocomputer device 206 through sensors measuring torque, operating time,and other data relevant to the particular power tool. Computer device206 compares the data received from the power tool with the dataspecified in the various assembly instructions for the product. Forexample, an assembly instruction may specify that screw gun 230 apply aspecific amount of torque to a fastener attached to a component. Ifcomputer device 206 detects the torque level is incorrect, in someembodiments, computer device 206 prevents or stops assembler 228 fromcontinuing until the torque level is corrected and the proper torqueapplied. Moreover, in some embodiments, a power tool may requireservicing or recalibration if the operating time of a tool such as screwgun 230 has exceeded a predetermined time or operating intervalthreshold.

Some embodiments store all the assembly information remotely overnetwork 222 using a network accessible server 224 and databases 226. Inthe example illustrated in FIG. 2, databases 226 includes at least anERP database 226 a, an assembly sequence database 226 b, an assemblertrain/certify database 226 c, and an assembly record database 226 d. ERPdatabase 226 a includes resource related data for all different aspectsof manufacturing and product assembly. For example, the data in ERPdatabase 226 a may be used in some embodiments to determine ifsufficient materials are available to complete a required number ofassemblies.

In addition to materials, ERP database 226 a has additional resourceinformation on trained assemblers qualified and ready to assemblespecified materials into products. This additional resource informationin ERP database 226 a may be cross-referenced by those assemblers inassembler train/certify database 226 c who have been trained to performthe assembly of certain products. In some embodiments, assemblers may beinitially qualified to assemble certain products but over time mayeventually need to be recertified and/or retrained. The time intervalallowed between retraining may be shorter if the particular assemblysequence is complex and more critical, while a longer time intervalbetween retraining may be allowed if the assembly sequence coversoperations considered more routine and less critical to the overallassembly.

If assembler 228 is trained, assembly sequence database 226 b providesthe sequence of assembly instructions to be followed when assembling aproduct. The workstation 124 presents instructions for assembling aproduct using a variety of multimedia data displayed through a userinterface on display device 214 and audio through audio speakers (notillustrated in FIG. 2). Likewise, the results of these assemblies arealso recorded using a variety of multimedia and then stored in assemblyrecord database 226 d for future reference and potential auditing.

Referring to FIG. 3, a flowchart diagram illustrates the operationsassociated with generating a sequence of assembly operations to beprocessed by a computer directed assembly workstation. Initially, insome embodiments, a sequence of assembly operations is identified forcombining materials into a component of the manufactured product (302).In some embodiments, a manufacture engineer or product designer alsocreates the sequence of assembly instructions during the design anddevelopment of the product. Over time, the sequence of assemblyoperations may be modified and refined to improve the product qualityand reduce associated assembly time.

Each operation may further specify a predetermined pocket in a matrixtray for holding a material to be used when an assembler performs thesequence of assembly operations (304). In some embodiments, the matrixtray is adapted to receive insertable shims with different pocket sizesfor holding various size materials. Different shims may be inserted intothe tray to accommodate the assembly of different materials as used inthe creation of various products. Generally, the person creating theassembly operations also specifies the configuration of the matrix trayincluding the number of insertable shims and associated pockets. Pocketsin the shims are arranged in a row and a column “matrix” configuration,and a specific pocket may be used by referencing a row and columnlocation on the overall matrix tray in accordance with some embodiments.For example, one matrix tray may be configured to receive four 2×2 shimseach having 4 individual pockets, the 16 pockets in this embodimentwould be addressed according to (X,Y) coordinates corresponding to therows and columns of a 2×8 matrix. Materials placed in the predeterminedpocket of a given matrix are generally taken from a Bill of Materials(BOM) stored in an enterprise database system, such as an ERP system,that manages inventory as products are manufactured. In someembodiments, materials in each pocket are consumed or used by at leastone assembly operation performed from the sequence of assemblyoperations. Since an exact amount of parts for an assembly should beplaced in the matrix tray, left over parts in the matrix tray afterassembly indicates that the product has likely been assembledincorrectly.

Next, in some embodiments, multimedia assembly data and related assemblyinstructions are generated for presentation through a user interface ofa computer device (306). Multimedia assembly data may include a varietyof images, video, text, and audio related to the materials, components,and final product being assembled and produced. Some embodiments ofmultimedia assembly data may include photos of specific fasteners orgroup of fasteners in the pocket of a matrix tray along with videos ofthe fasteners being inserted into a component of a product. Otherembodiments of the multimedia assembly data may include images of wirematerials held in a matrix tray along with additional images of thewires attached to an insertion point in the component of a product. Inaddition to images and videos, some embodiments of multimedia assemblydata may further include displaying text on the user interface of thecomputer that the assembler can read and follow. Computer directedassembly operations delivered through a combination of textinstructions, videos and images serve as a powerful guide for assemblerscombining materials into a component and product in accordance with someembodiments.

In some embodiments, the assembly instructions may determine whether anassembly instruction requests a tool to combine materials with thecomponent or product being assembled (308). If an assembly instructiondoes not request the use of a tool (308—No), the assembler may attachmaterials to a component directly with their hands. This might bepreferred if hand-tightening a fastener or other material is preferred.Alternatively, the assembly instruction may request using a tool(308—Yes) if the tool would aide in performing the assembly quickly andwith a higher degree of accuracy and quality. For example, a powerscrewdriver designed in accordance with some embodiments having abuilt-in torque sensor is useful if an assembly instruction requiresattaching multiple fasteners to a component at a predetermined torque.Manual tools such as screwdrivers and wrenches may also be used ifsensors, such as the torque sensor, are not required or a fastener neednot be attached as precisely as a power tool with sensors is capable.

To guide in the use of these tools, the assembly instruction generatedalso provides multimedia tool data on the user interface (310).Multimedia tool data may include a variety of images, video, text, andaudio related to the power tools or manual tools and their use inattaching fasteners or other materials during the sequence of assemblyoperations. Some embodiments of multimedia tool data may includedisplaying a schematic image or photo of a tool along with a textdescription of the tool on the user interface. The text description ofthe tool may also describe how the tool should be used to attach afastener or material or which torque setting should be used whentightening the fastener. If the assembler needs even more detailedguidance on using a tool, predetermined audio describing the use of thetool may accompany the images, videos, and other multimedia tool datadisplayed on the user interface.

Next, the assembly instruction in accordance with some embodimentsguides an assembler to perform the assembly operation that combines amaterial with a component used in creating a product (312). Inaccordance with some embodiments, the assembly instruction guides theassembler to perform one assembly operation from a sequence of assemblyoperations for assembling a product. For example, one assemblyinstruction may instruct an assembler to attach multiple fasteners, suchas several metal screws, from a first component to a second component ofa product. In this example, the assembly instruction is used to completethe assembly operation associated with attaching the first and secondcomponents together as one unit within the final product.

Next, the assembly instruction in some embodiments may optionallyinclude a verification operation to ensure materials and components areproperly combined together during the assembly (314). An assemblyinstruction may further request verification that materials andcomponents have been assembled together correctly (314—Yes). In someembodiments, the verification operation records multimedia verificationdata associated with a material as it is combined with an in-usecomponent of the finally assembled product (316). As previouslydescribed, the in-use component is the portion of the product currentlybeing worked on by the assembler. For example, one verification methodincludes recording images or video of the in-use component beingassembled or worked upon by the assembler and then storing the resultsin an assembly database for later review and/or analysis. In analternate approach to verification, the assembler places a wand having acolored tip or end near the portion of the in-use component beingassembled as an image or video of the area is recorded by a cameraassociated with the workstation. In some embodiments, the area near thetip of the wand is further analyzed to determine if the assemblerinstalled the materials in the correct area or portion of the in-usecomponent. In yet another embodiment, computerized comparisons areperformed of the images or videos of the in-use component beingassembled with predetermined images, videos, and/or other multimediadata associated with a previously assembled component and materials.

Alternatively, in some embodiments, the assembly instruction does notinclude a verification operation and no verification operation isperformed (314—No). When this occurs, (i.e., no verification operationis performed), the next assembly instruction in a sequence of assemblyinstructions is performed (304) and many of the above sequence of stepsin the flowchart of FIG. 3 are repeated.

FIG. 4 provides a flowchart diagram of the operations for specifyingmatrix trays and selectable inserts used with multimedia assembly datain some embodiments. To accommodate different materials during theassembly, the product designer or manufacture engineer specifies aparticular matrix tray design and the materials that should go in thepockets of the matrix tray (402). In some embodiments, the matrix trayis a combination of one or more interchangeable shim inserts tailored tothe particular assembly being performed. Each interchangeable shiminsert has one or more pockets arranged in rows and columns for holdingdifferent size and quantities of materials to be used during thesequence of assembly operations.

Generating the multimedia assembly data begins by creating an image ofthe matrix tray holding the materials to be used by the assemblerperforming the sequence of assembly operations (404). In someembodiments, some of the pockets in the matrix tray hold materials whileother pockets in the matrix tray are intentionally left empty. Next, theassembly instruction configures the user interface of the computerdevice to display the image of the matrix tray having materials storedin the various pockets (406). By displaying an image of the matrix trayand materials, the assembler can more readily locate the materials andperform the sequence of assembly operations. In some embodiments,highlighting a portion of the image of the matrix tray near thepredetermined pocket containing the material also serves to assist inlocating materials during the assembly process. For example, the portionof the image may be highlighted by further displaying a geometric shape,such as a square or circle, around the area of interest in the image ofthe matrix tray. Assembly instructions in some embodiments may also beconfigured to send a request through the user interface of the computerdevice asking the assembler to provide an indication of a pocket on anin-use matrix tray holding the material the assembler intends to use inconjunction with performing the at least one assembly operation. Thein-use matrix tray is the tray the assembler is using for the currentone or more assembly operations. For example, the assembler may use awand with a colored tip and move the tip of the wand over the area ofthe in-use matrix tray where the materials to be used for the assemblyare located.

In some embodiments, the assembly instruction may then instruct a cameraassociated with the assembler's workstation to acquire at least oneimage of the in-use matrix tray (408). The image taken of the in-usematrix tray in some embodiments should also include the indication fromthe assembler of the pocket on the in-use matrix tray holding thematerial to be used with the assembly operation. Image processing may beused in further embodiments to determine how much materials from theportion of the matrix tray have been used by the assembler and whethermore materials are left. If the materials in the matrix tray aredetermined not to match the expected amounts, a warning may be displayedon the user interface of the workstation indicating that some type oferror has occurred. Once the image has been processed, in someembodiments, the image of the in-use matrix tray is stored in anassembly record database creating a traceable record of the materialsused in performing the assembly operation (410).

FIG. 5 is another flow chart diagram outlining the training verificationoperations performed in accordance with some embodiments prior toproviding an assembler authorization to perform an assembly operation.Training verification operations may be performed in addition to theoperations associated with generating multimedia assembly data andassembly instructions in FIG. 3 at 306. In one embodiment, the trainingverification operations in FIG. 5 ensure that each person assembling aproduct (also referred to as an “assembler”) has been properly trainedand that products will be assembled properly with the highest qualityand zero defects. Accordingly, an assembly instruction may specify aprerequisite training sequence to be completed by each assembler beforeperforming the instruction presented through the user interface of aworkstation or computer device (502). In some embodiments, theprerequisite training sequence is the identical sequence of actions forthe particular assembly instruction and assembly operation. For example,if the assembly instruction is to install five (5) standoff screws in aprinted circuit board then the prerequisite training sequence may be toperform the identical task of installing five (5) standoff screws in aprinted circuit board with a power tool. Alternatively, similar tasks ofinstalling three (3) or more standoff screws in a printed circuit boardmay also satisfy the prerequisite training sequence for the assemblyinstruction.

Next, the assembly instruction may then request a training historyassociated with an assembler from a training database that includes aset of training sequences performed by the assembler (504). In certainembodiments, a detailed history of the various skills amassed by eachassembler is kept or stored in a training database. Some skills in thetraining database may be acquired when the assembler performs a trainingexercise while other skills in the training database may result when theassembler performs other assemblies and task. In some embodiments, eachassembly instruction may check the training database to determinewhether the assembler is trained to perform the particular instructionor task.

A determination is then made whether the training history associatedwith the assembler includes the specified prerequisite training sequence(506). For example, the training history for the assembler may alreadyinclude a task of installing five (5) standoff screws in a printedcircuit board with a power tool. Accordingly, in the event the assembleralready meets the specified prerequisite training sequence (506—Yes),some embodiments will then authorize the assembler to perform theinstruction displayed on the user interface and guide the assembler tocombine the materials from a pocket in a matrix tray with the component(510). Alternatively, the assembler may be required to perform aprerequisite training sequence when the determination indicates that theassembler has not been trained with the prerequisite training sequence(506—No). If this occurs, the assembler must first perform theprerequisite training sequence before proceeding with the assemblyinstruction and further operations to assemble the product (508). Oncethe assembler performs the prerequisite training sequence, in certainembodiments, the assembler is then authorized to perform the instructiondisplayed on the user interface (510).

FIG. 6A schematically illustrates an exemplary computer-implemented userinterface method for interacting with assembly operations in accordancewith some embodiments. Areas displayed on the user interface assist andguide an assembler through the assembly operations to create amanufactured product. These areas include assembly build informationarea 604, in-use view area 606, text assembly instructions area 608,validation area 610, preassembled view area 612, tool area view area614, and inventory area 616.

Assembly build information area 604 of the user interface detailsinformation on the product being assembled and the assembler currentlyperforming the assembly on the workstation. In this example, assemblybuild information area 604 provides a part number and details on theparticular release and version of the part number. In addition, theperson associated with assembling the particular product is identifiedas “Rob Jones”. Details from assembly build information area 604 arepermanently recorded in an assembly database along with other detailsassociated with the assembly of the components and this product.

In-use view area 606 is an area on the user interface that provides animage of an item the assembler is currently working on in conjunctionwith assembling the manufactured product. Some embodiments of the in-useview area 606 display live video recordings of the in-use item as theassembler positions the in-use item in front of a camera on theworkstation and performs a task in accordance with one of the assemblyinstructions. The in-use item may include one or any number of differentitems used by an assembler during the assembly. The in-use items in someembodiments may include a component making up a portion of themanufactured product, a matrix tray holding various materials, or amaterial, such as a fastener, to be attached to the component of theproduct being assembled.

In some embodiments, live video displaying the in-use item beingassembled may highlight a portion of the image to assist the assemblerin performing an assembly instruction. For example, in-use view area 606in FIG. 6A displays a component 606B lying on a workbench surface 606Aready for assembly. In this embodiment, a geometric shape such asrectangle highlight 606C overlays the image of component 606B andhighlights where materials from a matrix tray are to be attached. Tofurther assist the assembler, text assembly instructions area 608includes a sequence of assembly instructions for the assembly ofcomponent 606B into a product. In reference to the area underrectangular highlight 606C, exemplary text assembly instructionsspecified in assembly instruction 608A directs the assembler to, “STACKTWO STANDOFFS TOGETHER AND THEN INSTALL INTO CONTROLLER.” Additionally,assembly instructions area 608 in FIG. 6 include a created-by-entry 608Bthat stores the name of the person who created the assemblyinstructions—in this case, a J. WOOLISCROFT—and an assembly instructionsequence counter 608C that indicates the current assembly instruction is6 out of 23 instructions.

Preassembled view area 612 is an area on the user interface thatprovides an image of a preassembled item to guide the assembler in theassembly of the manufactured product. The preassembled item is anexemplary component assembled correctly in advance by a skilledassembler and now can be used as a model or example for the assembly ofnew items displayed in the in-use view area 606. In the illustratedexemplary embodiment, preassembled view area 612 includes a matrix tray612A assembled with insertable shim 612F having two pockets withfasteners, insertable shim 612G having twenty pockets with fasteners,and insertable shim 612J having twenty pockets with fasteners. Asillustrated, a portion of matrix tray 612A also has an open area 612Kthat is holding a strap fastener 612H ready to be assembled into acomponent or product.

Preassembled view area 612 further includes several other selectableviews in addition to tray view 612B that include detail view 1 612C,detail view 2 612D and detail view 3 612E. In some embodiments, aportion of the preassembled item may be highlighted in preassembled viewarea 612 to help the assembler find a material or attach a material tothe proper component. In one illustrated embodiment in FIG. 6A, matrixtray 612A has used rectangular highlight 612I to draw the attention tothe location of the standoffs referenced in the sequence of assemblyinstructions 608A in the assembly instruction view area 608 aspreviously described. In general, the preassembled item may include avariety of different items including a matrix tray assembled withmaterials (such as matrix tray 612A), a product assembled with one ormore components, a component assembled with one or more materials (suchas component 612E), and materials to be attached to the at least onein-use item the assembler is currently using (such as materials 612C).In accordance with some embodiments, the matrix tray 612A is onepreassembled item having several insertable shims and numerousfasteners.

Tool view area 614 on user interface 600 provides an image of a tool asspecified in the sequence of assembly instructions. The tool displayedin tool view area 614 is specified in the assembly instructions toassist the assembler in selecting the next tool to use in the subsequentsteps in assembling the manufactured product. For example, assemblyinstructions specify wand 614A as the tool to be used by the assemblerto assist in verifying that an assembly instruction has been performedcorrectly. Other tools that may be specified to appear in tool view area614 include manual tools and power tools as appropriate for theparticular assembly.

FIG. 6B illustrates an exemplary matrix tray and insertable shims forholding materials in accordance with some embodiments. Unlikeconventional material trays fixed in a single configuration, matrix tray618 can be tailored to accommodate the size and number of materialsrequired for each product. Different insertable shims allow both thesize and the number of pockets in matrix tray 618 to change to providesuitable capacity, yet keep a common form factor for ease of handlingand compatibility. Since the pockets in matrix tray 618 are kept inpredetermined rows and columns, assembly instructions reference specifictrays using a bar code of each tray and then a row/column (e.g., (x,y)coordinates) to address specific pockets in the tray. As an addedbenefit, this regular organization enables computers executing imageprocessing routines to more easily recognize the pockets, and thematerials kept in these pockets.

Database applications and embodiments may also identify and associatematerials with assembly instructions as the assembly instructions areperformed by the assembler. In the illustrated exemplary embodiment inFIG. 6B, each of the twenty (20) pockets from insertable shim 618C or618E may be digitally identified with (x,y) coordinates or enumerated aspockets 0 through 19 by one or more database programs. In someembodiments, image processing routines may capture images of matrix tray618 during assembly and quickly determine if the number of fasteners orother material for the assembly instruction are correct. To aid infurther identification and automation, some embodiments may individuallyidentify matrix tray 618 using a combination of one or several of a barcode (not shown), a QR code (not shown) or passive RFID technologyattached to matrix tray 618 (not shown).

In illustration of one embodiment, a tray portion 618A of matrix tray618 has been separated from insertable shims 618B, 618C, and 618E makingapparent the flexibility and accompanying advantages of the design. Trayportion 618A in some embodiments has four (4) areas labeled I, II, III,and IV adapted to receive either an insertable shim or directly receivematerial. For example, since strap fastener 618F cannot be containedwithin an insertable shim, it is instead placed directly on the area oftray portion 618A labeled III. As illustrated, insertable shims 618B,618C, and 618E may be replaced with different shims and inserted backonto tray 618A in different configurations as demanded by a differentset of assembly instructions and specific product being manufactured. Insome embodiments, shims may include two (2) pockets such as withinsertable shim 618B or twenty (20) pockets as exemplified by insertableshim 618C. Other shims may contain greater than two (2) pockets yetfewer than twenty (20) pockets (not shown) with the exact number ofpockets depending on the size and quantity of the materials used by aparticular assembly instruction or instructions. Alternate embodimentsmay also include tray portions larger than tray portion 618A asillustrated in FIG. 6B and formed from different geometric shapes otherthan a rectangle or other conventional geometries. It is also possiblethat a tray portion of a matrix tray has greater than just the four (4)areas illustrated in tray portion 618A; indeed the size of the trayportion 618A may be larger or smaller depending on the size of materialsbeing used and the application.

FIGS. 7A and 7B illustrate a flowchart diagram of an exemplary operationassociated with assembling materials into a component of a manufacturedproduct. In some embodiments, these flowchart operations are performedon a workstation as an assembler performs an assembly sequence ofinstructions to create the manufactured product. Initially, in someembodiments a training history is retrieved that is associated with anassembler selected to combine materials with a component of amanufactured product in accordance with a sequence of assemblyoperations (702). The training history may be stored in a trainingdatabase that describes the experience and/or training that an assemblerhas achieved through a combination of training assemblies and experienceassembling other components together into products.

Preferably, the assembler selected to assemble the product is alreadysufficiently trained and competent to perform each instruction of theassembly sequence. To make this determination, embodiments of thepresent invention check if the training history for the assemblersatisfies a prerequisite training sequence associated with the sequenceof assembly instructions (704). In one exemplary embodiment, theprerequisite training sequence may require the assembler to haveexperience or training installing a set of four standoffs with a powerscrewdriver before proceeding with the remaining instructions of theassembly sequence. This specific experience must be acquired throughtraining assemblies or practice assemblies before the assembler can makea production assembly for shipment.

If the assembler's training history does not include sufficientexperience (704—No), in certain embodiments, the assembler is guidedthrough a prerequisite training sequence using training multimedia dataon a user interface of a computer device at an assembly workstation(706). In some embodiments, the assembler is required to perform theactions associated with the training using sample materials and a samplecomponent. In some embodiments, training multimedia displayed on a userinterface of the workstation includes images of the materials andcomponents being assembled and text assembly instructions to guide theassembler through the training.

Once the assembler has performed the prerequisite training sequence, insome embodiments, the training history of the assembler is updated toinclude the prerequisite training sequence and associated assemblyperformance details (708). In some embodiments, the assembler's traininghistory is updated in a training database to include a speed and anaccuracy with which the assembly was performed. Speed and accuracyinformation helps determine how quickly an assembler is likely toperform a sequence of assembly instructions and also how many productsthe assembler is capable of assembling over a period of time. Thisperformance related information helps determine how to establish workschedules of people assembling certain products and meet productdelivery goals. For example, if a product delivery must take placequickly and with high quality results and zero-defects then only peoplewho have a history and are capable of assembling the product meetingthese constraints will be selected to assemble the product and fulfillthe order.

If the assembler's training history does include sufficient experience(704—Yes), in some embodiments, assembly instructions are providedthrough the user interface to guide the assembler in combining materialswith the component (710). To further assist the assembler, in someembodiments, specific materials stored in predetermine pockets of amatrix tray are identified using a variety of multimedia data andguidance is provided on assembling these materials with the component.

In some embodiments, assembly instructions determine whether a tool isused to combine materials with a component (712). If an assemblyinstruction does request using a tool (712—Yes), the assembler ispresented with a variety of multimedia tool data through the userinterface. The multimedia tool data guides the assembler in using thetool to perform a portion or all of the sequence of assemblyinstructions. In some embodiments, the tool multimedia data may be asingle image of a tool, a video of the tool, or a combination of imagesand video along with instructions for using the tool to assist inperforming the particular assembly instruction or instructions. Forexample, if the tool's usage is routine or typical then the multimediatool data may only need to identify the tool using a single imagedisplayed on the user interface. However, if the assembly instructionuses the tool in a more complicated manner, the multimedia tool data mayinclude images and videos along with detailed instructions for using thetool.

In some embodiments, the assembly instruction may not request using atool when the assembler can use their hands to perform an assembly orother task (712—No). For example, an assembly instruction may requestthat the assembler insert a wire connector with a group of wires into aconnector receiver on a printed circuit board. Referring to FIG. 7B,some embodiments may then determine if the assembly instructions shouldalso be verified using one or more verification operations (716). In theevent the assembler does not have to verify the assembly instruction(716—No), the above described steps in FIG. 7A starting with step 702are repeated until all the assembly instructions for the product arecompleted.

Alternatively, some embodiments may indeed require a verificationoperation to ensure the assembly instructions were performed correctly(714—Yes). In some embodiments, the verification operation recordsverification multimedia data associated with combining the materials andthe component (718). Once the assembler completes the assemblyinstruction or instructions, in certain embodiments, an image or videoof the in-use component being assembled is taken and stored in anassembly record database as a permanent record of the assembledcomponent or product. In further embodiments, the verification operationmay also request the user to point to or identify a location of theportion of the in-use component being assembled using a wand and thentake the images or videos as a permanent record. In another embodiment,image processing functions are utilized on a workstation to compare theimages of the in-use component with a predetermined multimedia assemblydata showing a previously assembled component or product known to beassembled correctly and with a high quality.

In accordance with some embodiments, FIG. 8 is a schematic block diagramof an exemplary computer device 800 capable of creating and processingassembly instructions for guiding assemblers manufacturing products.Computer device 800 includes a memory 802, presentation device driver804 coupled to a display device (not shown), a processor complex 806,secondary storage 808, network communication port 810 and I/O ports 812coupled to a variety of different input-output devices over aninterconnect 816. In particular, processor complex 806 may be a singleprocessor, multiple processors or multiple processor cores on a singledie. It is contemplated that processor complex 806 represents the one ormore computational units available in computer device 800. Further,input-output devices coupled to I/O ports 812 may include one or more ofthe following: cameras, power tools, power tools with sensors, wands,scanners, keyboards, mice, any other peripheral device previouslydescribed in conjunction with FIG. 2, and other suitable devices.Network communication port 810 may further include a WiFi, WiMAX orother connection to a network such as the Internet. Networkcommunication port 810 may also include wired connections to theInternet using CAT 5/6, Fiber Channel or similar approaches.

In the illustrative embodiment in FIG. 8, memory 802 includes storagelocations that are addressable by the processor complex 806 and adaptersfor storing software program code and data. For example, memory 802 mayinclude a form of random access memory (RAM) that is generally clearedby a power cycle or other reboot operation and classified as “volatile”memory. Processor complex 806 and various adapters may, in turn,comprise processing elements and logic circuitry configured to executethe software code and manipulate the data stored in the memory 802. Incomparison, secondary storage 808 may be a form of non-volatile storagefor storing a copy of run-time environment 826, applications and otherdata used by computer device 800. Alternatively, secondary storage 808may include conventional magnetic tapes or disks, optical disks such asCD-ROM, DVD, magneto-optical (MO) storage or any other type ofnon-volatile storage devices suitable for storing large quantities ofdata. These latter storage device types may be accessed locally througha direct connection to interconnect 816 or remotely in the “cloud”through network communication port 810 with an appropriate networkprotocol.

In some embodiments, memory 802 includes assembly instruction sequencecomponent 818, tool usage component 820, material usage component 822,multimedia assembly/verify component 824, and run-time module 826.Assembly instruction sequence component 818 includes methods and systemsfor creating assembly sequences for combining materials, components, andparts into products as described previously in conjunction with FIG. 3through FIG. 5. This assembly instruction sequence component 818 alsoprocesses these assembly instructions to guide and control assemblerscreating products also as previously described in conjunction with FIGS.7A-7B. Tool usage component 820 incorporates the creation andpresentation of multimedia tool data on a user interface to also assistin guiding an assembler creating products. As previously described inconjunction with FIG. 6B, material usage component 822 includes dataassociated with the configuration of matrix trays, insertable shims, andthe materials stored in these for the creation of different products.Multimedia assembly record/verify component 824 includes images, video,data, and processes for verifying the assembly of products as alsodescribed in conjunction with at least FIGS. 2-4 and FIGS. 7A-7B.Lastly, memory 702 includes run-time environment 826 portions of whichtypically reside in memory and are executed by the processing elements.Run-time environment 826 may be based upon a general-purpose operatingsystem, such as Linux, UNIX or Windows, the AppleOS or any othergeneral-purpose operating system as well as mobile or embedded operatingsystems based upon Android, Blackberry, QNX, Apple iOS, and others asused in mobile phones, mobile devices, touchpads, or touchscreen-basedcomputer systems.

While examples and implementations have been described, they should notserve to limit any aspect of the disclosure. Accordingly, embodimentscan be implemented in digital electronic circuitry, or in computerhardware, firmware, software, or in combinations of them. Apparatus ofthe disclosure can be implemented in a computer program product tangiblyembodied in a machine readable storage device for execution by aprogrammable processor; and method steps of the disclosure can beperformed by a programmable processor executing a program ofinstructions to perform functions by operating on input data andgenerating output. Embodiments can be implemented advantageously in oneor more computer programs that are executable on a programmable systemincluding at least one programmable processor coupled to receive dataand instructions from, and to transmit data and instructions to, a datastorage system, at least one input device, and at least one outputdevice. Each computer program can be implemented in a high levelprocedural or object oriented programming language, or in assembly ormachine language if desired; and in any case, the language can be acompiled or interpreted language. Suitable processors include, by way ofexample, both general and special purpose microprocessors. Generally, aprocessor will receive instructions and data from a read only memoryand/or a random access memory. Generally, a computer will include one ormore mass storage devices for storing data files; such devices includemagnetic disks, such as internal hard disks and removable disks; magnetooptical disks; and optical disks. Storage devices suitable for tangiblyembodying computer program instructions and data include all forms ofnon-volatile memory, including by way of example semiconductor memorydevices, such as EPROM, EEPROM, and flash memory devices; magnetic diskssuch as internal hard disks and removable disks; magneto optical disks;and CD ROM disks. Any of the foregoing can be supplemented by, orincorporated in, ASICs.

While specific embodiments have been described herein for purposes ofillustration, various modifications may be made without departing fromthe spirit and scope of the disclosure. Accordingly, the disclosure isnot limited to the above-described implementations, but instead isdefined by the appended claims in light of their full scope ofequivalents.

1. A computer-implemented user interface method displayed on a displaydevice of a computer device for interacting with a sequence of assemblyinstructions for creating a manufactured product, the method comprising:providing an in-use view area on the user interface that provides animage of at least one in-use item an assembler is currently using inconjunction with assembling a manufactured product; providing apreassembled view area on the user interface that provides an image ofat least one preassembled item used in guiding the assembler in theassembly of the manufactured product with the at least one in-use itemthe assembler is currently using; and providing a text assemblyinstructions area that provides at least one instruction from thesequence of assembly instructions for using the at least one in-use itemthe assembler is currently using in conjunction with assembling themanufactured product.
 2. The computer-implemented user interface methodof claim 1, wherein the in-use item is selected from a set of in-useitems including a component making up a portion of the manufacturedproduct, a matrix tray assembled with materials, and a material to beattached to the component.
 3. The computer-implemented user interfacemethod of claim 1, wherein the in-use item is being assembled inaccordance with the sequence of assembly instructions.
 4. Thecomputer-implemented user interface method of claim 1, wherein a portionof the image of the at least one in-use item is highlighted and thenidentified near where a material is to be attached to the in-use itembeing assembled.
 5. The computer-implemented user interface method ofclaim 1, wherein the at least one preassembled item is selected from aset of preassembled items including a matrix tray assembled withmaterials, a product assembled with one or more components, a componentassembled with one or more materials, and materials to be attached tothe at least one in-use item the assembler is currently using.
 6. Acomputer program product for creating a user interface for display on adisplay device of a computer device to interact with a sequence ofassembly instructions for creating a manufactured product, tangiblystored on a computer readable medium, comprising instructions operableto cause a programmable processor to: provide an in-use view area on theuser interface that provides an image of at least one in-use item anassembler is currently using in conjunction with assembling amanufactured product; provide a preassembled view area on the userinterface that provides an image of at least one preassembled item usedin guiding the assembler in the assembly of the manufactured productwith the at least one in-use item the assembler is currently using; andprovide a text assembly instructions area that provides at least oneinstruction from the sequence of assembly instructions for using the atleast one in-use item the assembler is currently using in conjunctionwith assembling the manufactured product.
 7. The computer programproduct of claim 6, wherein the in-use item is selected from a set ofin-use items including a component making up a portion of themanufactured product, a matrix tray assembled with materials, and amaterial to be attached to the component.
 8. The computer programproduct of claim 6, wherein the in-use item is being assembled inaccordance with the sequence of assembly instructions.
 9. The computerprogram product of claim 6, wherein a portion of the image of the atleast one in-use item is highlighted and then identified near where amaterial is to be attached to the in-use item being assembled.
 10. Thecomputer program product of claim 6, wherein the at least onepreassembled item is selected from a set of preassembled items includinga matrix tray assembled with materials, a product assembled with one ormore components, a component assembled with one or more materials, andmaterials to be attached to the at least one in-use item the assembleris currently using.
 11. An apparatus that creates a user interface fordisplay on a display device of a computer device to interact with asequence of assembly instructions for creating a manufactured product,the apparatus comprising: a processor capable of executing instructions;and a memory holding instructions that when executed by the processorcause the processor to: provide an in-use view area on the userinterface that provides an image of at least one in-use item anassembler is currently using in conjunction with assembling amanufactured product, provide a preassembled view area on the userinterface that provides an image of at least one preassembled item usedin guiding the assembler in the assembly of the manufactured productwith the at least one in-use item the assembler is currently using, andprovide a text assembly instructions area that provides at least oneinstruction from the sequence of assembly instructions for using the atleast one in-use item the assembler is currently using in conjunctionwith assembling the manufactured product.
 12. The apparatus of claim 11,wherein the in-use item is selected from a set of in-use items includinga component making up a portion of the manufactured product, a matrixtray assembled with materials, and a material to be attached to thecomponent.
 13. The apparatus of claim 11, wherein the in-use item isassembled in accordance with the sequence of assembly instructions. 14.The apparatus of claim 11, wherein a portion of the image of the atleast one in-use item is highlighted and then identified near where amaterial is to be attached to the in-use item being assembled.
 15. Theapparatus of claim 11, wherein the at least one preassembled item isselected from a set of preassembled items including a matrix trayassembled with materials, a product assembled with one or morecomponents, a component assembled with one or more materials, andmaterials to be attached to the at least one in-use item the assembleris currently using.